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FOREWORD 


During  the  last  couple  of  years  some  Defense  Logistics  Agency  (DLA)  supply  centers  have 
developed  procedures  whereby  competitive  contracts  are  awarded  to  other  than  low  bidders  if 
awardees  agree  to  provide  earlier  deliveries.  Delivery  Evaluation  Factors  (DEFs)  are  applied  to 
convert  reduced  Production  Lead  Times  (PLTs)  to  dollar  values.  These  are  used  to  adjust  bid 
prices  for  evaluation  purposes. 

The  Defense  Industrial  Supply  Center  has  taken  the  lead  in  developing  procedures  for  using 
DEFs.  It  has  been  using  the  system  for  small,  competitive  purchases  for  stock  since  February 
1992.  Recently  it  expanded  the  program  to  include  large  purchases.  Somewhat  similar 
programs  have  been  introduced  at  the  Defense  Construction  Supply  Center  and  the  Defense 
General  Supply  Center. 

The  three  programs  differ  in  many  respects,  including  methods  of  determining  evaluation 
adjustments  for  bids  having  different  PLTs,  resulting  incentive  payments  to  vendors,  types  of 
buys  covered,  and  implementation  procedures.  Also  concerns  have  been  raised  regarding  the 
technical  merits  of  using  DEFs.  DLA's  Procurement  Policy  Group  (AQPL)  desired  to  have  an 
objective  analysis  of  the  technical  merits,  costs,  and  benefits  of  the  various  DEF  programs. 

This  report  shows  the  effects  that  the  program  has  had  in  terms  of  PLTs,  unit  prices  paid  to 
vendors,  and  late  deliveries.  Results  for  National  Stock  Numbers  for  which  DEF  buys  were  used 
are  compared  with  results  for  similar  NSNs  with  no  DEF  buys.  The  evaluation  adjustment 
procedures  used  by  different  supply  centers  are  compared.  The  validity  of  the  various  technical 
concerns  is  analyzed.  Finally  recommendations  for  improvements  in  the  program  are  made. 


Chief,  DLA  Operations  Research  Office 
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INSIGHT  THROUGH  ANALYSIS 


EXECUTIVE  SUMMARY 


During  the  last  couple  of  years  three  Defense  Logistics  Agency  (DLA)  supply  centers  have 
developed  Delivery  Evaluation  Factor  (DEF)  procedures  whereby  competitive  contracts  are 
awarded  to  other  than  low  bidders  in  return  for  early  deliveries.  The  three  programs  differ  in 
several  major  respects,  including  methods  of  calculating  evaluation  adjustments,  types  of  buys 
covered,  and  implementation  procedures.  Concerns  have  been  raised  regarding  both  the  overall 
value  of  the  program  and  the  relative  merits  of  the  individual  programs  at  each  center. 

This  study  provides  an  analysis  of  the  technical  merits,  costs,  and  benefits  of  the  various  DEF 
programs.  Much  of  the  analysis  is  based  upon  a  review  of  the  contract  files  for  Defense 
Industrial  Supply  Center  (DISC)  for  the  period,  January  1992  through  June  1994.  Experience 
with  the  DEF  program  at  the  other  supply  centers  is  too  limited  for  meaningful  analysis.  The 
study  also  compares  the  evaluation  adjustment  procedures  used  by  the  various  supply  centers. 

Average  Production  Lead  Times  (PLTs)  declined  during  the  period  analyzed.  The  DEF  program, 
as  well  as  a  number  of  other  programs,  probably  contributed  to  this  decline.  DEF  buys  displayed 
especially  large  reductions  in  both  average  PLTs  and  percentages  of  late  deliveries. 
Unfortunately,  these  reductions  were  not  completely  sustained  in  subsequent  nonDEF  buys.  This 
is  disturbing  since  the  theoretical  justification  for  paying  incentives  for  lower  PLTs  depends 
upon  their  being  sustained  in  subsequent  nonDEF  buys. 

The  analysis  did  not  show  that  DEF  buys  cause  increases  in  average  unit  prices.  The  method  of 
determining  evaluation  adjustments  used  by  the  Defense  Construction  Supply  Center  (DCSC) 
has  better  theoretical  justification  and  results  in  lower  incentive  payments  than  does  the  method 
used  by  the  Defense  General  Supply  Center  and  DISC.  However,  the  DCSC  method  also  has 
some  limitations  concerning  the  use  of  Federal  Supply  Class  (FSC)  aggregate  factors  for  specific 
National  Stock  Number  items.  Item  variability  within  a  FSC  could  result  in  excessive  or 
insufficient  adjustments. 

Insufficient  information  exists  for  making  overall  judgments  about  the  extent  to  which  DEF 
programs  should  be  used.  The  answer  depends  upon  considerations  unique  to  each  supply 
center,  such  as  buyer  and  contractor  awareness  of  the  value  of  consistently  low  PLTs  and  use  of 
other  programs  to  maintain  low  PLTs. 

When  use  of  DEFs  is  appropriate,  evaluation  adjustments  should  be  determined  in  accordance 
with  a  new  proposed  method.  This  method  combines  the  theoretical  base  of  the  DCSC  method 
with  item  unique  data  to  more  accurately  estimate  the  value  of  reduced  PLT. 
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SECTION  1 
INTRODUCTION 


1.1  BACKGROUND 

During  the  last  couple  of  years  some  supply  centers  have  developed  procedures  whereby 
competitive  contracts  are  awarded  to  other  than  the  low  bidders  if  awardees  agree  to  provide 
earlier  deliveries.  Delivery  Evaluation  Factors  (DEFs)  are  applied  to  convert  reduced  Production 
Lead  Times  (PLTs)  to  dollar  values.  These  are  used  to  adjust  bid  prices  for  evaluation  purposes. 

The  Defense  Industrial  Supply  Center  (DISC)  has  taken  the  lead  in  developing  procedures  for 
using  DEFs.  It  has  been  using  the  system  for  small,  competitive  purchases  for  stock  since 
February  1992.  In  June  1994,  it  expanded  the  system  to  include  large  purchases.  The  Defense 
Construction  Supply  Center  (DCSC)  has  been  using  a  somewhat  similar  program  for  large 
purchases  since  April  1994.  Its  program  is  limited  to  National  Stock  Numbers  (NSNs)  that  have 
been  determined  to  have  artificially  inflated  PLTs.  The  Delivery  Reduction  Program, 
implemented  at  the  Defense  General  Supply  Center  (DGSC)  in  March  1994,  also  is  generally 
similar  to  the  DISC  program.  It  is  used  for  stock  buys  over  $10,000.  The  three  programs  differ 
in  many  respects,  including  methods  of  calculating  evaluation  adjustments,  types  of  buys 
covered,  and  implementation  procedures. 

Several  concerns  have  been  raised  regarding  the  technical  merits  of  using  DEFs.  These  include: 
(1)  since  unplanned  early  deliveries  do  not  immediately  reduce  safety  stocks,  the  assumed 
benefits  may  not  be  realized;  (2)  DEFs  may  need  to  be  modified  for  very  early  deliveries  because 
of  diminishing  marginal  benefits;  (3)  early  deliveries  expected  in  DEF  procurements  may  not  be 
achieved;  (4)  although  expected  PLT  reductions  may  be  achieved  in  initial  deliveries,  they  may 
not  be  sustained  in  subsequent  deliveries,  thereby  voiding  the  expected  benefits;  and  (5)  the 
various  methods  of  calculating  DEFs  may  not  be  equally  appropriate. 

Beginning  in  1987  the  DLA  Operations  Research  Office  (DORO)  has  conducted  a  series  of 
studies  on  costs  of  late  contractor  deliveries  for  supply  centers.  The  most  recent  one  is  entitled, 
"Cost  of  Late  Delivery  for  Post  Award  Consideration",  DLA-94-P301 16,  May  1994.  It  was 
thought  that  some  of  the  concepts  and  data  developed  in  these  studies  may  be  useful  in 
estimating  the  effects  of  early  deliveries  for  use  in  developing  DEFs. 

1.2  OBJECTIVE 

The  overall  objective  of  this  study  is  to  provide  a  sound  analysis  of  the  costs  and  benefits  of  the 
various  DEF  programs.  Specifically,  the  objectives  are:  to  evaluate  the  use  of  DEFs  by  supply 
centers  to  ascertain  their  effects  on  PLTs;  and  to  analyze  the  technical  merits  of  the  use  of  DEFs 
and  determine  what  improvements,  if  any,  should  be  made. 
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1.3 


SCOPE 


The  study  addresses  the  use  of  DEFs  at  DCSC,  the  Defense  Electronics  Supply  Center  (DESC), 
DGSC,  and  DISC.  Analyses  of  data  in  the  contract  files  are  confined  to  DISC  for  the  period, 
January  1992  through  June  1994.  The  other  supply  centers  have  had  insufficient  experience 
with  the  DEF  program  for  meaningful  analysis. 


1-2 


SECTION  2 
METHODOLOGY 


2.1  OVERVIEW 


The  results  presented  in  this  report  are  derived  primarily  from  analysis  of  DEF  policies  and 
procedures  as  provided  by  personnel  at  headquarters  DLA  and  the  supply  centers  and  by  review 
of  the  contract  files.  These  are  discussed  in  subsections  2.3  and  2.4,  respectively.  In  addition, 
concepts  and  data  provided  in  the  related  earlier  DORO  reports  mentioned  in  the  Introduction 
were  used. 

It  should  be  pointed  out  that  DLA  has  a  number  of  other  programs  which  may  also  tend  to  reduce 
lead  time.  It  is  difficult  to  separate  the  effects  of  the  DEF  program  from  the  other  ongoing 
programs.  These  include: 

•Automated  Best  Value  Model  (ABVM) 

•Quality  Vendor  Program  (QVP) 

•Blue  Chip  Vendor  Program  (BCVP) 

•Buy  Response  Vice  Inventory  (BRVI) 

•Standard  Automated  Material  Management  System  (SAMMS)  Purchased  by  Electronic 
Data  Exchange  (SPEDE) 

•Paperless  Order  Processing  System  (POPS) 

•Commodity  Oriented  Purchasing  System  (COPS) 

•Electronic  Commerce 
•Electronic  Bid  Board 

The  following  subsection  provides  some  technical  definitions  used  throughout  this  report. 

2.2  DEFINITIONS 

2.2.1  EVALUATION  ADJUSTMENT 

The  term,  "evaluation  adjustment",  refers  to  the  difference  between  total  evaluated  price  after 
applying  a  DEF  adjustment  for  delivery  earlier  or  later  than  requested  and  total  price  as  quoted. 
Other  names  include  "evaluation  factor  adjustment"  (DCSC),  "SAdjustment"  (DGSC),  and 
"delivery  evaluation  factor"  (DISC). 

2.2.2  EVALUATION  INCENTIVE 

The  term,  "evaluation  incentive",  refers  to  the  amount  paid  for  a  PLT  reduction,  provided  that 
delivery  is  in  accordance  with  the  contract.  It  is  the  difference  between  the  low  bidder's  total 
unevaluated  price  and  the  awardee's  total  unevaluated  price.  Other  names  include  "price 
differential  paid"  (DCSC),  "delivery  reduction  incentive"  (DGSC),  and  "total  evaluation 
differential"  (DISC).  Normally  it  is  less  than  or  equal  to  the  evaluation  adjustment. 
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2.2.3 


DEF  BUYS 


DEF  buys  are  purchases  that  involve  payment  of  evaluation  incentives.  Buys  which  go  through 
the  DEF  evaluation  process  but  do  not  involve  payment  of  evaluation  incentives  are  not 
included.  This  occurs  when  application  of  evaluation  incentives  are  not  needed,  (e.g.  when  a 
bidder  offers  both  the  lowest  price  and  the  shortest  delivery)  or  when  the  differences  in  bids  are 
greater  than  the  DEF  evaluation  adjustments. 

2.2.4  NONDEF  BUYS 

NonDEF  buys  are  all  purchases  that  are  not  DEF  buys. 

2.3  DEF  POLICIES  AND  PROCEDURES 

Documents  referenced  in  this  section  are  available  upon  request. 

2.3.1  HEADQUARTERS  DLA 

Personnel  at  headquarters  (AQP)  provided  background  and  general  information  on  the  DEF 
program  both  telephonically  and  during  visits.  Important  guidance  was  provided  at  an  In 
Process  Review  on  1  September  1994.  In  addition,  methodology  concerns  were  reviewed  with 
MMS  concerning  the  validity  of  savings  associated  with  lead  time  reductions. 

2.3.2  DCSC 

Personnel  at  DCSC  provided  information  on  its  program  telephonically.  Written  materials  also 
were  provided.  The  key  documents  reviewed  are  the  IOMs  on  "Delivery  Evaluation  Factor  Test 
Procedures",  dated  12  August  1993,  and  "DCSC  Proposed  Production  Leadtimes  (PLT) 
Evaluation  Factor  Clause  and  Provision",  dated  22  April  1993. 

2.3.3  DESC 

Personnel  at  DESC  provided  information  telephonically.  DESC  does  not  have  a  DEF  program. 

2.3.4  DGSC 

A  meeting  with  personnel  at  DGSC  was  held  on  8  June  1994.  Written  materials  also  were 
provided.  The  key  document  reviewed  is  Policy  Information  Memorandum  94-8,  dated  25 
March  1994. 


2.3.5  DISC 

A  meeting  was  held  with  personnel  at  DISC  on  7  July  1994.  Notes  on  this  meeting  were  faxed 
to  points  of  contact  at  each  of  the  supply  centers  included  in  the  study  on  15  July  1994. 
Personnel  at  DISC  also  provided  information  and  comments  telephonically  and  in  written 
materials.  The  key  document  reviewed  is  the  memorandum  on  "Expanding  the  Use  of  the 
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Delivery  Evaluation  Factor  (DEF)  Program  to  Large  Purchases  (DPPT  94-4)",  dated  21  April 
1994. 


2.4  CONTRACT  FILES 

2.4.1  GENERAL 


Analyses  of  data  in  the  contract  files  regarding  the  effectiveness  of  DEF  buys  were  confined  to 
DISC.  Experience  with  DEF  buys  at  the  other  supply  centers  is  either  nonexistent  or  so  limited 
and  recent  as  to  be  of  little  or  no  analytical  value. 

DEF  buys  are  distinguished  from  nonDEF  buys  in  the  contract  files  by  the  Procurement 
Instrument  Identification  Numbers  (PIINs).  Purchase  orders  for  DEF  buys  have  "W"  in  the 
ninth  position.  Purchase  orders  for  DEF  buys  awarded  based  on  the  Blue  Chip  Vendor  Program, 
also  known  as  the  Quality  Vendor  Program,  have  "WB"  in  the  ninth  and  tenth  positions. 
Contracts  for  DEF  buys  have  "CW"  in  the  ninth  and  tenth  positions. 

Separate  analyses  were  made  of  National  Stock  Numbers  (NSNs)  with  DEF  buys  and 
comparable  NSNs  with  no  DEF  buys.  Both  analyses  cover  the  period,  January  1992  through 
June  1994.  The  following  statistics  were  calculated: 

1.  Average  PLT,  defined  as  ship  date  less  award  date.  Ship  dates  were  used  in 
lieu  of  receipt  dates  because  they  are  more  reliable.  Receipt  dates  contained  in  the  contract  files 
often  substantially  lag  actual  receipts.  Both  arithmetic  means  and  medians  were  computed. 

2.  Average  price  per  unit.  Both  arithmetic  means  and  medians  were  computed. 

3.  Percentage  of  buys  that  were  delivered  late.  Late  is  defined  as  ship  date  later 
than  contract  delivery  date. 

For  the  purpose  of  determining  the  above  statistics  each  buy  was  given  equal  weight  (as  opposed 
to  weighting  each  NSN  equally  or  weightings  proportional  to  quantities  or  dollar  values). 

Where  averages  were  expressed  both  as  medians  and  as  arithmetic  means,  greater  reliance  was 
placed  on  the  medians.  This  is  because  they  are  less  influenced  by  abnormal  outliers. 

2.4.2  DEF  BUYS 

The  analysis  of  NSNs  with  DEF  buys  covers  only  NSNs  which  had  their  first  DEF  buy  in 
Calendar  Year  (CY)  1993  and  which  also  had  at  least  one  nonDEF  buy  during  the  period  of 
analysis.  DEF  buys  after  CY  93  were  eliminated.  This  was  done  to  determine  the  effects  of 
DEF  buys  compared  with  preDEF  buys  and  to  determine  whether  any  effects  persist  in 
subsequent  nonDEF  buys.  The  latter  is  of  crucial  importance  because  (according  to  inventory 
control  theory)  payment  of  evaluation  incentives  is  justified  only  if  PLT  reductions  persist  in 
subsequent  nonDEF  buys.  In  order  to  reduce  distortion  associated  with  purchase  quantities,  all 
buys  for  an  NSN  that  involved  purchase  quantities  differing  by  more  than  a  factor  of  five  (i.e. 


2-3 


over  500  percent  or  less  than  20  percent)  of  the  average  DEF  buy  for  that  NSN  during  CY  93 
were  eliminated.  Statistics  were  calculated  separately  for  preDEF  buys,  DEF  buys,  and  postDEF 
buys. 


2.4.3  NO  DEF  BUYS 

The  analysis  of  NSNs  with  no  DEF  buys  covers  only  NSNs  in  Federal  Supply  Classes  (FSCs) 
that  had  DEF  buys  during  CY  93.  This  was  done  to  provide  reasonable  comparability  of  NSNs 
included  in  the  two  analyses.  In  order  to  reduce  distortion  associated  with  purchase  quantities, 
all  buys  for  an  NSN  that  involved  purchase  quantities  differing  by  more  than  a  factor  of  five 
from  the  average  buy  for  that  NSN  during  CY  93  were  eliminated.  Statistics  were  calculated 
separately  for  1992, 1993,  and  1994  (six  months). 

2.4.4  COMPARISONS 

Although  the  two  analyses  are  intended  to  represent  NSNs  differing  only  in  the  use  of  DEF  buys, 
comparisons  are  not  as  clear  as  one  would  like.  In  particular,  DEF  buys  are  always  competitive. 
Most  nonDEF  buys  are  not.  Furthermore,  if  nonDEF  buys  are  competitive,  they  usually  go 
through  the  DEF  evaluation  process  but  are  given  to  low  bidders  without  the  need  to  apply 
evaluation  incentives. 


2.4.5  SUPPLEMENTARY  ANALYSES 

Certain  other  analyses  of  the  contract  files  also  were  made.  The  more  important  of  these  are: 

1 .  A  ten  percent  random  sample  was  taken  of  all  NSNs  which  had  their  first  DEF 
buy  in  CY  93  and  also  nonDEF  buys  during  the  period  of  analysis.  For  each  buy  in  each  NSN 
the  award  date,  contract  PLT,  actual  PLT,  type  of  buy,  unit  price,  and  purchase  quantity  was 
shown  for  the  entire  period  of  analysis.  This  analysis  served  a  variety  of  purposes  including  help 
in  designing  the  analyses  described  in  previous  paragraphs  of  this  section. 

2.  An  analysis  of  the  distribution  of  cycle  times  for  all  NSNs  at  each  of  the  four 
supply  centers  as  of  the  second  quarter  of  CY  94  was  made.  This  helped  in  relating  potential 
PLT  reductions  to  procurement  cycles,  which  were  used  in  comparing  alternative  methods  of 
determining  evaluation  adjustments. 
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SECTION  3 
RESULTS 

3.1  STATISTICS  PROVIDED  BY  DISC 

The  results  of  the  DORO  analyses  are  shown  in  subsequent  subsections.  Relevant  statistics 
provided  by  DISC  are  shown  below.  They  were  derived  from  the  contract  files  for  small 
purchases  for  Fiscal  Year  (FY)  1993  and  FY  1994  through  March. 

Table  3-1 

Statistics  Provided  bv  DISC 


r - tt:  itt:  ^ 


FY  93 

FY  94 

Total  number  of  awards 

126,000 

59,000 

DEF  buys  as  %  of  total  awards 

4.1% 

5.7% 

Average  PLT,  all  buys  (days) 

199 

204 

Average  PLT,  DEF  buys  only  (days) 

89 

105 

i  ' 


Although  the  DEF  program  was  generally  used  throughout  the  period,  only  a  small  portion  of  the 
buys  involved  payment  of  evaluation  differentials.  Average  PLTs  were  much  lower  for  DEF 
buys  than  for  all  buys. 

The  DISC  analysis  shown  above  should  not  be  directly  compared  with  the  DORO  analysis 
shown  in  the  subsequent  subsections.  The  methodology  used  in  the  two  analyses  are  quite 
different.  For  example,  the  DISC  analysis  includes  all  NSNs,  whereas  the  DORO  analysis  is 
limited  to  NSNs  in  FSCs  which  had  initial  DEF  buys  during  CY  93.  Also  the  measures  of  PLTs 
used  in  the  two  analyses  are  different.  DISC  used  receipt  date  less  award  date.  DORO  used  ship 
date  less  award  date.  The  difference  between  ship  date  and  receipt  date  includes  not  only  transit 
time  but  also  any  delay  at  the  receiving  depot  for  processing  and  recording  receipts. 

3.2  NUMBER  OF  OBSERVATIONS 

3.2.1  DEF  BUYS 

The  DORO  analysis  of  NSNs  with  DEF  buys  is  based  upon  the  following  numbers  of  buys: 
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Table  3-2 

Number  of  Observations:  Analysis  of  NSNs  with  DEF  Buys 


3.2.2  NO  DEF  BUYS 

The  analysis  of  comparable  NSNs  with  no  DEF  buys  is  based  upon  the  following  numbers  of 
buys: 


Table  3-3 


3.2.3  EVALUATION 

The  numbers  of  observations  are  sufficiently  large  to  provide  valid  comparisons. 
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3.3 


PI  T  COMPARISON 


3.3.1  DEF  BUYS 

The  average  PLTs,  in  calendar  days,  of  NSNs  purchased  using  DEF  buys  are  as  follows: 


Table  3-4 

Average  PLTs:  NSNs  with  DEF  Buys 


PreDEF  DEF  PostDEF 


NSNs  with  nonDEF  buys 

Median 

95 

54 

before  and  after  DEF  buys. 

Mean 

111 

66 

NSNs  with  no  nonDEF  buys 

Median 

98 

57 

after  DEF  buys. 

Mean 

118 

69 

NSNs  with  no  nonDEF  buys  Median  54 

before  DEF  buys.  Mean  71 


73 

88 


3.3.2  NO  DEF  BUYS 

The  average  PLTs  of  comparable  NSNs  not  purchased  using  DEF  buys  are  as  follows: 


Table  3-5 


Average  PLTs:  NSNs  with  No  DEF  Buys 


3.3.3 


COMPARISON 


Average  PLTs  declined  during  the  period  of  analysis.  The  DEF  program,  as  well  as  a  number  of 
other  programs,  probably  contributed  to  this  decline.  Having  a  DEF  program  in  place,  tangibly 
demonstrates  the  importance  management  places  on  short  PLTs. 

DEF  buys  displayed  especially  large  reductions  in  average  PLTs.  Unfortunately,  the  reductions 
were  not  completely  sustained  in  subsequent  nonDEF  buys.  Roughly  one-third  of  the  PLT 
reduction  was  not  sustained  (a  "rebound"  effect).  Overall  percentage  declines  in  average  PLTs 
were  about  the  same  regardless  of  whether  any  DEF  buys  had  been  used.  Average  PLTs  were 
higher  for  nonDEF  buys  of  NSNs  with  DEF  buys  than  for  NSNs  with  no  DEF  buys.  This 
probably  reflects  a  tendency  to  use  DEFs  for  NSNs  that  display  long  PLTs. 

3.4  UNIT  PRICE  COMPARISON 

3.4.1  DEF  BUYS 

The  average  unit  prices  of  NSNs  purchased  using  DEF  buys  are  as  follows: 

Table  3-6 

Average  Unit  Prices:  NSNs  with  DEF  Buys 


Unit  Price 


PreDEF 

DEF 

PostDEF 

NSNs  with  nonDEF  buys 
before  and  afterDEF  buys. 

Median 

Mean 

$6.35 

48.54 

$5.41 

48.53 

$5.01 

44.90 

NSNs  with  no  nonDEF  buys 
after  DEF  buys. 

Median 

Mean 

4.96 

32.50 

4.26 

30.80 

NSNs  with  no  nonDEF  buys 
before  DEF  buys. 

Median 

Mean 

4.32 

31.56 

4.45 

33.15 

V _ J 


3.4.2  NO  DEF  BUYS 

The  average  unit  prices  of  comparable  NSNs  not  purchased  using  DEF  buys  are  as  follows: 
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Table  3-7 


1992 

Unit  Price 

1993  1994 

NSNs  with  buys  in 

CY92,  93,  &  94. 

Median 

Mean 

$  17.92 
130.05 

$  19.26 
149.19 

$  14.36 
121.68 

NSNs  with  buys  only 
in  CY  92  &  93. 

Median 

Mean 

11.60 

105.01 

10.92 

97.54 

NSNs  with  buys  only 
in  CY  93&  94. 

Median 

Mean 

15.51 

93.46 

11.70 

78.30 

V _ J 

Average  Unit  Prices:  NSNs  with  No  DEF  Buys 


3.4.3  COMPARISON 

Average  unit  prices  declined  slightly  during  the  period  of  analysis.  There  is  no  indication  that 
this  is  related  to  the  DEF  program.  The  analysis  does  not  show  that  DEF  buys  cause  increases  in 
average  unit  prices. 

3.5  LATE  DELIVERY  COMPARISON 

3.5.1  DEF  BUYS 

The  percentages  of  buys  that  were  delivered  late  for  NSNs  purchased  using  DEF  buys  are  as 
follows: 


Table  3-8 

Late  Deliveries:  NSNs  with  DEF  Buys 


Late  Deliveries  ('%') 

PreDEF 

DEF  PostDEF 

NSNs  with  nonDEF  buys  before 
and  after  DEF  buys. 

28.5 

15.4  26.7 

NSNs  with  no  nonDEF  buys 
after  DEF  buys. 

32.0 

14.8 

NSNs  with  no  nonDEF  buys 
^before  DEF  buys. 

13.5  33.8 
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3.5.2 


NO  DEF  BUYS 


The  percentages  of  buys  that  were  delivered  late  for  comparable  NSNs  not  purchased  using  DEF 
buys  were  as  follows: 


Table  3-9 

Late  Deliveries:  NSNs  with  No  DEF  Buys 


Late  Deliveries  (%} 


1992 

1993 

1994 

NSNs  with  buys  in  CY  92, 

93,  &  94. 

18.7 

16.8 

12.0 

NSNs  with  buys  only  in 

CY  92  &  93. 

23.4 

22.5 

NSNs  with  buys  only  in 

CY  93  &  94. 

18.5 

11.9 

3.5.3  COMPARISON 

The  percentages  of  late  deliveries  were  highest  for  nonDEF  buys  of  NSNs  which  had  some  DEF 
buys.  This  suggests  that  DEF  buys  tended  to  be  used  for  NSNs  which  had  long  PLTs  and  were 
more  likely  to  be  delivered  late.  Use  of  DEF  buys  substantially  reduced  late  deliveries. 
Unfortunately,  the  improvements  were  not  sustained  in  subsequent  nonDEF  buys. 

3.6  EVALUATION  ADJUSTMENTS 

3.6.1  DCSC 

The  evaluation  adjustments  used  at  DCSC  basically  are  the  products  of  the  following  three 
terms: 


1 .  Day  Cost  Ratio  for  applicable  FSC.  This  is  cost  per  day  of  lateness  of 
additional  inventory  resulting  from  increased  projected  PLT  triggered  by  late  delivery,  expressed 
as  a  decimal  fraction  of  contract  price.  The  cost  of  the  additional  inventory  is  the  present  value 
of  the  interest  and  storage  costs  required  to  carry  the  additional  safety  stock  over  the  useful  life  of 
the  item.  The  latest  estimates  of  these  costs  are  contained  in  the  DORO  report  referenced  in 
Section  1.1  of  this  report. 

2.  Number  of  days  early  (maximum  of  30;  negative  numbers  for  days  late). 
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3.6.2 


3.  Total  contract  price  in  dollars. 
DGSC  &  DISC 


The  evaluation  adjustments  used  by  DGSC  and  DISC  are  essentially  the  same  (although  the 
procedures  used  for  forecasting  demand  differ).  They  are  the  products  of  the  following  two 
terms: 


1 .  Average  daily  demand  in  dollars. 

2.  Number  of  days  early  (negative  numbers  for  days  late). 

3.6.3  COMPARISON 

The  two  methods  can  be  directly  compared.  The  rationale  is  difficult  to  explain  in  simple 
sentences.  It  may  be  explained  algebraically  as  follows: 

Let: 

R  =  Day  Cost  Ratio 
E  =  Number  of  days  early 
P  =  Total  contract  price  in  dollars 
D  =  Average  daily  demand  in  dollars 
C  =  Procurement  cycle  in  days 

Y  =  Evaluation  adjustment  used  by  DCSC 

Z  =  Evaluation  adjustment  used  by  DGSC  &  DISC 
X  =  Ratio  of  evaluation  adjustments  =  Y/Z 

Then  from  the  descriptions  of  the  evaluation  adjustments  for  DCSC  and  for  DGSC  &  DISC 
given  in  Sections  3.6.1  and  3.6.2: 

Y  =  R  *  E  *  P 
Z  =  D  *  E 

In  order  to  compute  the  ratios  between  the  two  methods  of  determining  evaluation  adjustments, 
the  expressions  given  above  for  Y  and  Z  may  be  substituted  in  the  expression  for  X: 

X  =  Y/Z 

=  (R  *  E  *  P)  /  (D  *  E) 

=  (R  *  P)  /  D 

The  total  contract  price  is  equal  to  the  procurement  cycle  times  the  average  daily  demand.  Hence: 


P  =  C  *  D 
D  =  P  /  C 


3-7 


Substituting  for  D  in  the  last  expression  for  X: 


X  =  (R  *  P)  /  (P  /  C) 

=  (R  *  P  *  C)  /  P 
=  R*C 

Using  the  latest  estimates  of  the  overall  average  Day  Cost  Ratios  (shown  in  the  report  referenced 
in  Section  1.1)  and  the  median  procurement  cycle  times  (as  of  the  second  quarter  of  CY  94),  the 
ratios  between  the  two  methods  of  determining  evaluation  adjustments  may  be  computed  for 
each  supply  center.  The  results  are: 


Table  3-10 

Comparison  of  Methods  of  Determining  Evaluation  Adjustments 


DCSC 

DESC 

DGSC 

DISC 

Overall  Average  Day  Cost  Ratio  =  R 

.00134 

.00043 

.00081 

.00060 

Median  Procurement  Cycle  (days)  =  C 

365 

578 

426 

365 

Ratio  of  Evaluation  Adjustments 
(DCSC  method  /  DGSC  &  DISC  method) 

0.49 

0.25 

0.35 

0.22 

V _ _ _ J 


The  method  of  determining  evaluation  adjustments  used  by  DCSC  results  in  substantially 
smaller  adjustments  than  does  the  method  used  by  DGSC  and  DISC,  ranging  from  22  percent  to 
49  percent  and  averaging  about  1/3  of  the  amounts  using  the  DGSC  and  DISC  method.  Of 
course  the  evaluation  incentives  paid  by  DCSC  are  correspondingly  smaller. 

3.6.4  ADVANTAGES  AND  DISADVANTAGES 

Both  current  methods  are  very  easy  to  apply.  However,  each  method  has  drawbacks  that  could 
result  in  protests,  excessive  or  insufficient  evaluation  factor  being  applied  and/or  non-optimal 
awards.  The  main  disadvantage  of  the  DCSC  method  is  that  the  Day  Cost  Ratio  was  developed 
as  an  average  for  an  entire  FSC  or  Center.  Its  use  for  a  specific  NSN  could  result  in  errors 
depending  on  how  far  from  average  the  particular  NSN  behaves  concerning  costs  associated  with 
lead  time  changes.  The  DISC  (and  DGSC)  method  uses  item  dependent  factors  rather  than 
averages.  However,  inventory  control  theory  refutes  the  direct  link  between  lead  time  reduction 
and  inventory  levels.  The  buffer  effect  of  a  variable  safety  level  to  prevent  backorders  results  in 
an  indirect  effect.  It  can  be  shown  mathematically  that  only  the  variable  safety  level  quantity  is 
affected  by  lead  time  changes  and  that  the  reorder  quantity  is  set  only  by  demand  and 
procurement  cycle  time.  Thus  if  PLT  is  reduced,  there  will  only  be  a  permanent  reduction  in 


3-8 


safety  levels  and  overall  inventory  levels  are  reduced  by  the  safety  level  reduction.  Lower  PLT 
can  delay  future  awards,  but  the  amount  ordered  can  only  be  reduced  by  the  safety  level 
reduction  associated  with  the  PLT  reduction  of  the  prior  buy.  Since  the  DCSC  factors  are  based 
solely  on  safety  level  effect,  the  theoretical  underpinning  of  the  method  is  sound.  The  other 
method  is  simple  and  seems  to  make  sense,  but  in  reality  overstates  the  impact  of  the  shorter 
PLT. 


3.6.5  AN  ALTERNATIVE  APPROACH 

In  an  effort  to  capture  the  strengths  of  each  of  the  two  existing  methods,  a  third  alternative 
method  was  developed.  Like  the  DISC  method  it  avoids  average  factors  and  uses  item  specific 
data.  Like  the  DCSC  method  it  is  based  on  safety  level  changes  as  a  basis  for  savings  associated 
with  PLT  reductions.  In  fact,  this  approach  assumes  a  linear  relationship  between  PLT  and 
variable  safety  level.  Also,  this  approach  recognizes  the  fact  that  PLT  reductions  may  not  be 
completely  sustained  in  future  buys,  the  "rebound"  effect.  In  addition  to  being  more  complex, 
this  approach  requires  some  research  during  bid  evaluation.  The  proposed  DEF  is  shown  below: 

PROPOSED  DEF 


Definitions: 

Cj  -  The  unit  price  of  the  low  bid 

C2  -  The  unit  price  of  the  evaluated  bid  i.e.  early  delivery 

ALT-  Administrative  Lead  Time  of  record 
PLT-  Production  Lead  Time  of  record 

DAYS  -  Difference  in  delivery  days  between  evaluated  bid  and  low  bid 
QTY  -  quantity  of  purchase 

VSL  -  Variable  Safety  Level  from  Supply  Control  Study 

k-  constant  to  reflect  inabilty  to  sustain  PLT  reduction  (rebound  effect) 

Award  to  low  bid  if: 

C2  QTY+VSL 

-  >  - 

Cj  (  QTY+  j  ALT+  PLT-fDAYS*kt  j*VSL  ) 

ALT+PLT 


Rationale: 

This  approach  assumes  VSL  is  directly  proportional  to  PLT  reductions.  The  SAMMS 
formula  is  complex  but  simulation  models  have  shown  that  effect  is  nearly  linear.  If  lead  time  is 
zero,  VSL  is  zero;  and  if  DAYS  is  zero,  VSL  is  unchanged.  This  formula  is  derived  from  the 
concept  that  bids  are  equal  if  the  unit  price  times  the  inventory  affected  are  the  same.  If  C2  is 
greater  than  C, ,  the  total  cost  can  still  be  less  since  it  will  be  applied  against  a  lower  total 
quantity  because  the  VSL  is  reduced.  The  value  for  the  constant  k  could  be  set  to  activate  when 
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DAYS  exceeds  some  high  number  set  by  the  Center  reflecting  higher  likelihood  of  a  rebound  for 
extreme  early  deliveries.  Data  from  DISC  suggests  that  the  value  for  k  should  be  about  2/3. 
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SECTION  4 
CONCLUSIONS 


4.1  OVERVIEW 

Conclusions  are  divided  into  two  groups.  Section  4.2  addresses  the  five  specific  concerns  listed 
in  Section  1.1  of  this  report  (as  well  as  the  Task  Order  and  Study  Plan).  Section  4.3  provides 
other,  more  general  conclusions. 


4.2  CONCERNS 

4.2.1  CONCERN  1 

CONCERN:  Since  unplanned  early  deliveries  do  not  immediately  reduce  safety  stocks,  the 
assumed  benefits  may  not  be  realized. 

CONCLUSION:  This  concern  is  valid.  Deliveries  on  initial  DEF  buys  normally  are  unplanned 
in  the  sense  that  the  buys  are  triggered  by  PLTs  longer  than  the  actual  PLTs.  Thus,  material  is 
received  before  it  is  needed,  even  for  safety  stock.  Unneeded  early  deliveries  trigger  early 
payments.  Hence,  from  a  cash  flow  standpoint,  unneeded  early  deliveries  from  DEF  buys  are 
undesireable. 


4.2.2  CONCERN  2 

CONCERN:  DEFs  may  need  to  be  modified  for  very  early  deliveries  because  of  diminishing 
marginal  benefits. 

CONCLUSION:  This  concern  is  partly  valid.  There  is  no  supporting  theoretical  or  empirical 
evidence  that  suggests  diminishing  marginal  benefits  from  reduced  lead  times,  provided  that  the 
system  works  as  intended  (i.e.  that  although  evaluation  incentives  are  used  only  to  obtain  initial 
early  deliveries,  the  PLT  reductions  are  sustained  in  subsequent  buys).  On  the  contrary, 
inventory  theory  indicates  that  safety  stocks  are  almost  directly  proportional  to  lead  times. 
Savings  in  the  cost  of  maintaining  safety  stocks  per  day  of  reduced  lead  time  do  not  vary  with  the 
size  of  the  reduction.  Thus,  as  lead  times  are  reduced,  cost  savings  increase  correspondingly. 
Unfortunately,  empirical  evidence  suggests  that  the  system  often  does  not  work  as  intended  .  In 
particular,  PLT  reductions  in  DEF  buys  are  not  consistently  sustained  in  subsequent  nonDEF 
buys  (see  Concern  4). 


4.2.3  CONCERN  3 

CONCERN:  Early  deliveries  expected  in  DEF  procurements  may  not  be  achieved. 

CONCLUSION:  This  concern  is  not  valid.  The  analysis  indicates  that  about  85  percent  of  the 
DEF  buys  were  shipped  on  time.  This  is  actually  better  than  the  on  time  rates  for  nonDEF  buys. 
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4.2.4 


CONCERN  4 


CONCERN:  Although  expected  PLT  reductions  may  be  achieved  in  initial  deliveries,  they  may 
not  be  sustained  in  subsequent  deliveries,  thereby  voiding  the  expected  benefits. 

CONCLUSION:  This  concern  is  valid.  The  analysis  indicates  that  PLT  reductions  for  DEF 
buys  are  not  consistently  sustained  in  subsequent  nonDEF  buys.  Over  the  period  of  analysis, 
overall  percentage  declines  in  average  PLTs  were  about  the  same  regardless  of  whether  any  DEF 
buys  had  been  used.  This  is  especially  disturbing,  since  (according  to  inventory  control  theory) 
payments  of  evaluation  incentives  are  justified  by  the  safety  stock  reductions  made  possible  by 
projected  future  PLT  reductions. 

4.2.5  CONCERN  5 

CONCERN:  The  various  methods  of  calculating  DEFs  may  not  be  equally  appropriate. 

CONCLUSION:  This  concern  is  valid.  The  method  of  calculating  evaluation  adjustments  used 
by  DCSC  is  similar  to  the  method  used  by  most  supply  centers  for  determining  the  cost  of  late 
deliveries.  It  is  based  upon  well  accepted  inventory  control  theory  which  applies  to  changes  in 
projected  lead  times,  regardless  of  whether  the  changes  are  positive  (longer  PLTs)  or  negative 
(shorter  PLTs).  However,  the  factor  used  is  a  center  average  which  may  or  may  not  reflect  the 
costs  associated  with  a  specific  NSN.  The  method  used  by  DGSC  and  DISC  appears  to  be 
arbitrary.  It  results  in  substantially  higher  evaluation  incentives.  Considering  the  danger 
associated  with  using  an  average  factor  in  lieu  of  item  specific  data,  the  more  conservative 
proposed  method  outlined  in  3.6.5  is  preferable. 

4.3  GENERAL 

4.3.1  SUSTAINED  PLT  REDUCTIONS 

Although  the  percentage  declines  in  average  PLTs  were  about  the  same  regardless  of  whether 
DEF  buys  were  used,  one  cannot  be  certain  that  DEF  buys  do  not  result  in  PLT  reductions  in 
future  nonDEF  buys.  During  the  period  of  analysis,  PLTs  declined  across  the  board.  This  may 
be  the  result  of  using  the  DEF  program,  thereby  making  both  buyers  and  contractors  generally 
more  cognizant  of  the  importance  of  short  PLTs. 

4.3.2  UNIT  PRICES 

Although  payments  of  evaluation  incentives  tangibly  add  to  costs,  the  analysis  does  not  show 
that  DEF  buys  cause  increases  in  average  unit  prices. 
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4.3.3 


OVERALL  EVALUATION 


The  overall  effort  to  reduce  PLTs  is  working.  Because  of  the  number  of  programs  that  can 
affect  PLTs,  that  are  going  on  simultaneously,  it  is  difficult  to  separate  the  contributions  of  each 
program. 

This  study  does  not  provide  sufficient  information  for  making  overall  judgements  about  the 
extent  to  which  a  DEF  program  should  be  used.  The  answer  depends  upon  such  intangibles  as 
management  style,  buyer  and  contractor  awareness  of  the  value  of  consistently  low  PLTs,  other 
programs  to  maintain  low  PLTs,  etc.  The  best  answer  may  depend  upon  considerations  unique 
to  a  supply  center,  or  even  to  a  specific  business  unit  within  a  supply  center. 


4-3 


SECTION  5 

RECOMMENDATIONS 


Determination  of  when  use  of  evaluation  incentives  is  appropriate  should  be  left  to  the  discretion 
of  supply  center  management.  When  their  use  is  appropriate,  evaluation  adjustments  should  be 
determined  in  accordance  with  the  proposed  method  developed  in  3.6.5.  Because  of  data 
limitations,  it  would  be  appropriate  to  reevaluate  the  use  of  DEF's  when  all  centers  have  used  the 
technique  for  a  period  of  time,  perhaps  in  one  to  two  years. 
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